Glasses-type information device, and method and storage medium for the same

ABSTRACT

A glasses-type information device configured to execute an application includes at least one memory storing instructions, and at least one processor executing the instructions causing the glasses-type information device to execute display control in such a manner that an image corresponding to an operation unit relating to remote control on the application is projected in a field of vision of a user wearing the glasses-type information device by using the glasses-type information device, the remote control being executed by a linked device held by the user, and execute function control of the application based on a command received from the linked device in accordance with a user operation performed by the user on the linked device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of International Patent ApplicationNo. PCT % JP2022/004993, filed Feb. 9, 2022, which claims the benefit ofJapanese Patent Application No. 2021-024691, filed Feb. 18, 2021, bothof which are hereby incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a technique relating to a glasses-typeinformation device, such as a smart glass.

Background Art

Wearable terminals (smart glass, smart watch, smartphone, etc.) havebecome more diversified. Users often simultaneously wear a plurality ofterminals, and technologies of linking these wearable terminals havebeen proposed. Among these technologies, there has been proposed atechnique of displaying partial image data indicating a partial regionof an image being displayed on a smartphone, on a head-mounted display(HMD), in cooperation between the smartphone and the HMD (JapanesePatent Application Laid-Open No. 2019-36914).

CITATION LIST Patent Literature

-   PTL 1: Japanese Patent Laid-Open No. 2019-36914

In the days ahead, glasses-type information devices, such as smartglasses, might be routinely worn and used in replacement of smartphones.On the other hand, a wearable terminal, such as a smart watch, differentfrom a glasses-type information device is expected to continue to beused in combination with the glasses-type information device. It isconsidered to become necessary in the future to enhance usability inlinking these devices and remotely controlling the functionality of theglasses-type information device from the different wearable terminal.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, a glasses-typeinformation device configured to execute an application includes atleast one memory storing instructions, and at least one processorexecuting the instructions causing the glasses-type information deviceto execute display control in such a manner that an image correspondingto an operation unit relating to remote control on the application isprojected in a field of vision of a user wearing the glasses-typeinformation device by using the glasses-type information device, theremote control being executed by a linked device held by the user, andexecute function control of the application based on a command receivedfrom the linked device in accordance with a user operation performed bythe user on the linked device.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of a configuration of asystem including a glasses-type information device and another wearableterminal.

FIG. 2A is a diagram illustrating an example of a hardware configurationof smart glasses.

FIG. 2B is a diagram illustrating an example of a hardware configurationof a smart watch.

FIG. 3A is a diagram illustrating an example of a software moduleconfiguration of the smart glasses.

FIG. 3B is a diagram illustrating an example of a software moduleconfiguration of the smart watch.

FIG. 4 is a flowchart illustrating processing to be executed by awearable terminal that cooperates with a glasses-type information deviceaccording to a first exemplary embodiment.

FIG. 5 is a flowchart illustrating processing to be executed by aglasses-type information device that cooperates with another wearableterminal according to the first exemplary embodiment.

FIG. 6A illustrates an example of an initial screen on each device.

FIG. 6B illustrates an example of display control executed on eachdevice at the time of remote control start.

FIG. 6C illustrates an example of display control regarding pairingwhich is executed at the time of remote control start.

FIG. 7A illustrates an example of display control which is executed oneach device at the time of control application selection for remotecontrol.

FIG. 7B illustrates an example of display control which is executed oneach device during remote control.

FIG. 8A illustrates an example of first display control which isexecuted on each device when mirroring display is performed in remotecontrol.

FIG. 8B illustrates an example of display control which is executed oneach device when an operation performed during mirroring display isreflected.

FIG. 9A illustrates an example of second display control which isexecuted on each device when mirroring display is performed in remotecontrol.

FIG. 9B illustrates an example of display control which is executed oneach device when an operation performed during mirroring display isreflected.

FIG. 10A illustrates an example of first display control which isexecuted on each device in a case where mirroring display is notperformed in remote control.

FIG. 10B illustrates an example of second display control which isexecuted on each device in a case where mirroring display is notperformed in remote control.

FIG. 10C illustrates an example of third display control which isexecuted on each device in a case where mirroring display is notperformed in remote control.

FIG. 11 is a flowchart illustrating environment change detectionprocessing to be executed by a glasses-type information device thatcooperates with a wearable terminal according to a second exemplaryembodiment.

FIG. 12 is a flowchart illustrating processing to be executed by thewearable terminal that cooperates with the glasses-type informationdevice according to the second exemplary embodiment, when an environmentchange is detected.

FIG. 13 is a flowchart illustrating processing to be executed by theglasses-type information device that cooperates with the wearableterminal according to the second exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the drawings.

First Exemplary Embodiment

FIG. 1 illustrates an example of a configuration of a system including aglasses-type information device and a wearable terminal that cooperateswith the glasses-type information device. The glasses-type informationdevice is able to communicate with a mobile network (mobilecommunication network) 130 and the Internet 140 directly or via a mobilerouter (not illustrated). In the present invention, smart glasses 101are used as an example of the glasses-type information device. Theglasses-type information device is not limited to binocular smartglasses, and a monocular smart glass and a head-mounted display is alsoapplicable. The wearable terminal is an information device including anotification function such as a display function and/or a vibrationfunction, and a communication function such as Bluetooth® communicationthrough which communication with the glasses-type information device isenabled. In the present invention, a smart watch 110 is used as anexample of the wearable terminal.

The smart glasses 101 are a wearable information terminal to be worn bya user in the vicinity of his/her eyes. The smart glasses 101 display,on a display unit 102, a video which is a virtual image, in a field ofvision of the user without blocking the field of vision. Such a displaymethod is referred to as augmented reality (AR) or mixed reality (MR),and is provided by a function of projecting information onto atransmissive display (lens 105) or a retina (not illustrated) of theuser. FIG. 1 illustrates the smart glasses 101 that provide video to oneeye, but in the present invention, the smart glasses 101 that projectvideo in fields of vision of both eyes is also applicable.

The display unit 102 is also provided with a camera unit 103 forcapturing an image of a subject in an eye direction of the user. Anoperation frame 104 serves as a frame of the smart glasses 101, includesa touch sensor, and is used for operating a terminal. The operationframe 104 has a built-in speaker function, and can transmit sound to theuser.

The smart glasses 101 is capable of implementing an internally-storedmodule, such as an embedded Subscriber Identity Module (eSIM), and iscapable of connecting to the Internet 140 via the mobile network 130that uses a fourth Generation Mobile Communication System (4G) or fifthGeneration Mobile Communication System (5G) network. The smart glasses101 is also capable of connecting to a mobile router owned by the user,via a connection based on Wi-Fi or a universal serial bus (USB), andconnecting to the mobile network 130 via the mobile router. The smartglasses 101 is also capable of connecting to the Internet 140 via Wi-Ficonnection without going through the mobile network 130.

The smart watch 110 is a wristwatch-type information terminal to be wornby the user on his/her wrist. A display unit 111 not only displaysinformation such as a time but also functions as a touch panel withwhich the user is enabled to operate a terminal using the display unit111. Wireless communication 120 is used for performing datacommunication between the smart glasses 101 and the smart watch 110, andis assumed to be wireless communication complying with the Bluetooth®standard, but the wireless communication 120 is not limited to this. Thesmart watch 110 also includes a notification function based on avibration function, aside from a display function.

FIGS. 2A and 2B are diagrams each illustrating an example of a hardwareconfiguration of the smart glasses 101 and the smart watch 110,respectively, according to an exemplary embodiment of the presentinvention.

FIG. 2A is a diagram illustrating a hardware configuration of the smartglasses 101.

A central processing unit (CPU) 201 comprehensively controls variousfunctions of the smart glasses 101 via an internal bus 206 based onprograms stored in a read-only memory (ROM) 203. A result of executionof a program performed by the CPU 201 is projectable and displayable bya display 202 in a field of vision of the user as video. In the presentexemplary embodiment, a method by which the user views an imageprojected by the display 202 in the field of vision via the transmissivelens 105 is assumed as a display method. Nevertheless, a method by whichthe display 202 directly projects an image onto a retina is alsoemployable. The ROM 203 is, for example, a flash memory, and storesvarious types of setting information and application programs asdescribed above. A random access memory (RAM) 204 functions as a memoryor a work area of the CPU 201. The processing illustrated in aflowchart, which features the present exemplary embodiment, and will bedescribed below with reference to the drawings, is implemented by theCPU 201 executing a corresponding program.

A network interface (LF) 205 is a network module for connecting to themobile network 130 and Wi-Fi network. By this module, the smart glasses101 become connectable to the Internet 140. In a case where a mobilerouter is used, the smart glasses 101 can also establish connectionusing a USB I/F (not illustrated) of the smart glasses 101.

An operation unit 207 receives an input made by the user using theoperation frame 104, and transmits a signal corresponding to the input,to the above-described processing units via an operation unit I/F 208. Asensor unit 209 schematically illustrates one or more sensors. Morespecifically, at least any of a global positioning system (GPS), a gyrosensor, an acceleration sensor, a proximity sensor, and a bloodpressure/heart rate measurement sensor is mounted on the smart glasses101. In addition, a sensor for detecting biological information forimplementing fingerprint authentication, vein authentication, irisauthentication, and/or other authentications may be mounted on the smartglasses 101. A camera 210 has an image capturing function, and imagedata obtained by image capturing is stored into the ROM 203. A laser 211projects various pieces of content onto the display 202, or directlyprojects various pieces of content onto a retina for a retina projectionmethod. A storage device 212 is a storage medium, and is a devicestoring various types of data, such as applications. The storage device212 also includes a device for reading out data in a storage medium, anda device for deleting data in a storage medium. Some terminals includeonly the ROM 203, without including the storage device 212. A near fieldcommunication I/F 213 is used in communication with the smart watch 110,and realizes the wireless communication 120, for example.

While not illustrated, the smart glasses 101 is further configured toinclude a structure for realizing a voice call using a network or aphone line, so that the smart glasses 101 is usable in place ofmodern-day smartphones. More specifically, the smart glasses 101includes a component for connecting to a phone line, a speaker, amicrophone, a voice control chip, and the like.

FIG. 2B is a diagram illustrating a hardware configuration of the smartwatch 110.

A CPU 221 comprehensively controls various functions of the smart watch110 via an internal bus 226 based on programs stored in a ROM 223. Aresult of execution of an application program performed by the CPU 221,and a notification from the smart glasses 101, and other types ofinformation are displayed on a display 222. The ROM 223 is a flashmemory, and stores various types of setting information and applicationprograms as described above. A RAM 224 functions as a memory or a workarea of the CPU 221. The processing illustrated in a flowchart, whichfeatures the present exemplary embodiment, and will be described belowwith reference to the drawings, is implemented by the CPU 221 executinga corresponding program.

A near field communication I/F 225 performs control for the smart watch110 performing one-way or two-way data communication with an externalnetwork device via the wireless communication 120. An operation unit 227receives an input made by the user on the display unit 111, andtransmits a signal corresponding to the input, to the above-describedprocessing units via an operation unit LF 228. A configuration in whichthe display 222 and the operation unit 227 are integrated using a touchpanel is also implementable. A sensor unit 229 schematically illustratesone or more sensors. More specifically, at least any of a GPS, a gyrosensor, an acceleration sensor, a proximity sensor, and a bloodpressure/heart rate measurement sensor is mounted on the smart watch110. In addition, a sensor for detecting biological information forimplementing fingerprint authentication, vein authentication, irisauthentication, or other authentications may be mounted on the smartwatch 110.

As described above, the smart glasses 101 is capable of furtherexecuting software for implementing a voice call function.

FIGS. 3A and 3B are diagrams each illustrating an example of a softwaremodule configuration of the smart glasses 101 and the smart watch 110,respectively, according to an exemplary embodiment of the presentinvention.

FIG. 3A illustrates a software module configuration of the smart glasses101.

A communication unit 301 transmits instruction information indicating aninstruction to the smart watch 110, and status information about thesmart glasses 101 to the smart watch 110. In addition, the communicationunit 301 not only receives instruction information transmitted from thesmart watch 110, but also receives a notification from an applicationserver that cooperates with the smart glasses 101 via a network usingthe network I/F 205. The status information indicates whether the smartglasses 101 are currently used by the user, and indicates modeinformation (standby mode, power saving mode, etc.) for the smartglasses 101. A storage unit 302 performs information exchange betweenthe ROM 203 and the RAM 204, and other processing units. The storageunit 302 also stores image data obtained through image capturingexecuted by the camera 210.

A display unit 303 displays results of various applications executed bythe CPU 201, by, for example, AR or MR display. For example, using an ARtechnique, the display unit 303 displays content (projects information)onto the display 202 in such a manner that virtual content issuperimposed onto a real space. In the present invention, the method ofdirectly projecting content onto a retina is also employable. In theinformation projection, the position and orientation in the real spaceof the smart glasses 101 may be estimated based on a technique such assimultaneous localization and mapping (SLAM). An estimation result isused in display processing for virtual content.

An operation unit 304 receives an input made by the user using theoperation frame 104, receives instructions for operations of variousapplications and operations on the display on the display 202, andperforms control. A camera control unit 305 controls activation, stop,and image capturing of the camera 210, and stores image data obtained byimage capturing, into the ROM 203 in cooperation with the storage unit302. A mode control unit 306 controls mode switching between a pluralityof modes provided in the smart glasses 101, and switches the state ofthe smart glasses 101 to a mode desired by the user. A status detectionunit 307 manages information regarding a status change or a use statusthat has been detected by the sensor unit 209, and stores theinformation into the ROM 203. A pairing unit 308 controls the near fieldcommunication I/F 213 to enable two-way data communication with anexternal network device via the wireless communication 120.

An application management unit 309 manages execution statuses of variousapplications, version information about the various applications, anotification received from a server, and information indicating whethera power saving mode is on, and manages these pieces of information inthe ROM 203 and the RAM 204 via the storage unit 302. In the powersaving mode (including a sleep mode), control is performed such thatpower consumption is minimized in a case where a battery remainingamount of the smart glasses 101 is equal to a predetermined value orless.

The application management unit 309 also controls various applicationsbased on an instruction input by the user via the operation unit 304 oran instruction received from a linked (paired) external terminal via thecommunication unit 301. In the present exemplary embodiment, the linkedexternal terminal is a wearable terminal worn by the user wearing thesmart glasses 101, and a specific example of the wearable terminal isthe smart watch 110.

In some cases, the application management unit 309 receives either oneof the instruction input by the user via the operation unit 304 and theinstruction received from the external terminal via the communicationunit 301 earlier, and receives the other one later. In such a case, theinstruction received earlier is given a higher priority than to the onereceived later. Alternatively, the application management unit 309 maypredetermine which instruction is to be given a higher priorityuniformly or for each application. Yet alternatively, the instruction tobe given a higher priority may be made settable by the user.

In a case where content is displayed onto the display 202 at the time ofapplication control, the application management unit 309 displayscontent onto the display 202 via the display unit 303.

The software module configuration of the smart glasses 101 according tothe present exemplary embodiment has been described above in conjunctionwith FIG. 3A. The smart glasses 101 may include additionally a modulefor a common operation unit that is capable of commonly receivingoperation instructions independently of linked terminals, withoutindividually receiving operation instructions from various linkedterminals. Such operation instructions include instructions in acooperative function, such as an instruction to operate an applicationon the smart glasses 101 by remote control, and a display/hideinstruction for mirroring display. The definition of mirroring displaywill be described below.

Standardized instruction information common to the common operation unitand the linked terminals may be prepared, and the linked terminals andthe smart glasses 101 may exchange instruction information.Alternatively, the common operation unit may generate or prestore anoperation instruction user interface (UI) common to linked terminals,and distribute the operation instruction UI to the linked terminals, andthen, each linked terminal may display the distributed operationinstruction UI, receive a user operation, and notify the smart glasses101 of received information.

FIG. 3B illustrates a software module configuration of the smart watch110.

A clock unit 321 manages and controls time information to be displayedon the display 222. The management and control of time information maybe executed by an application program stored in the ROM 223. Acommunication unit 322 transmits instruction information to the smartglasses 101, and receives not only status information for the smartglasses 101 but also a notification from an application server. Astorage unit 323 performs information exchange between the ROM 223 andthe RAM 224, and other processing units. A display unit 324 displaystime information managed by the clock unit 321, and results of variousapplications executed by the CPU 221. The display unit 324 also displaysan application notification transferred from the smart glasses 101. Anoperation unit 325 receives an input made by the user on the displayunit 111, and controls operations of various applications and displayonto the display 222. A status detection unit 326 manages, asinformation, change in a status detected by the sensor unit 229, andstores the information into the ROM 223. A pairing unit 327 controls thenear field communication I/F 225 to enable two-way data communicationwith an external network device via the wireless communication 120.

An application management unit 328 manages execution statuses of variousapplications, version information about the various applications, anotification received from a server, and information indicating whetherthe state of the smart watch 110 is a sleep mode, and manages thesepieces of information in the ROM 223 and the RAM 224 via the storageunit 323. The smart watch 110 transitions to the sleep mode in a casewhere an operation on the smart watch 110 via the operation unit 325 isnot performed for a predetermined time or more. In the sleep mode,screen off and screen lock of the display 222 are performed.

The application management unit 328 also controls various applicationsbased on an input made by the user via the operation unit 325. In a casewhere content is displayed onto the display 222 when the applicationmanagement unit 328 controls an application, the application managementunit 328 displays content onto the display 222 via the display unit 324.

The application management unit 328 further managesapplication-to-application glasses cooperation information forpredetermining functions to be enabled when various applications on thesmart watch 110 and the smart glasses 101 operate in cooperation, in theROM 223 and the RAM 224.

The cooperation here is assumed to be remote control of an applicationon the smart glasses 101 that is to be executed by the smart watch 110.At the time of remote control, as a part of an image projected in afield of vision of the user that is provided by the smart glasses 101,simplified display of an operation board corresponding to the smartwatch 110 is provided. In the present exemplary embodiment, thesimplified display of an operation board corresponding to a wearableterminal, such as the smart watch 110, that is to be used in remotecontrol is also called mirroring display.

The application-based glasses cooperation information predetermines, foreach application, ON/OFF indicating whether to enable a remote controlfunction, and ON/OFF indicating whether to enable a mirroring displayfunction (refer to Table 1).

TABLE 1 Application-based glasses cooperation information ApplicationApplication Remote Mirroring ID Name Control Display 0001Application0001 ON ON 0002 Application0002 ON ON 0003 Application0003OFF OFF

A setting value of the ON/OFF indicating whether to enable mirroringdisplay may be included in install information for various applicationson the smart watch 110, or may be made settable by the user using asetting application (not illustrated) that is managed by the applicationmanagement unit 328.

On the premise of the software module configurations of the smartglasses 101 and the smart watch 110, which have been described above inconjunction with FIGS. 3A and 3B, a cooperative function of the smartglasses 101 and the smart watch 110, which is a characterizing featureof the exemplary embodiment, will be described. The followingdescription is premised on an operation to be executed for anapplication for which both remote control and mirroring display are setto ON in the above-described application-based glasses cooperationinformation (Table 1).

The smart glasses 101 of the present exemplary embodiment has the remotecontrol function of receiving user instruction information forcontrolling an application on the smart glasses 101, from a linkeddevice (the smart watch 110 in the present exemplary embodiment), andexecuting processing in such a manner that the application provides afunction based on the user instruction information. The smart glasses101 further receive, as user instruction information, an input made bythe user via the operation unit 325 on the smart watch 110, to directlycontrol a control target application on the smart glasses 101. Thesepieces of user instruction information sometimes include an applicationID of an application on the smart glasses 101 that indicates a controltarget application.

In addition, the smart glasses 101 has a function of an operating system(OS) for issuing a control instruction to a control target applicationon the smart glasses 101, and/or a dedicated control application. Thisfunctionality of the OS or the dedicated control application enablesdata corresponding to entered user instruction information to bedelivered to the control target application, so that the control targetapplication is controlled as instructed by the user. Examples of thecontrol application include an application corresponding to respectiveapplications of the smart glasses 101, and an application that is usedin common and is specialized in a dedicated function, such as anapplication dedicated for character input or an application dedicatedfor screen scroll. In addition, for example, when the controlapplication ends, user instruction information to be transmitted to thesmart glasses 101 is capable of include information indicating a controlend of a target application. In a case where the application managementunit 328 detects that a predetermined time has elapsed since the stateof the smart watch 110 transitions to the sleep mode, informationindicating the end of a control on a target application may be includedas user instruction information.

The smart glasses 101 has the mirroring display function of receiving UIinformation for a linked device (the smart watch 110) to be operated atthe time of remote control, and projecting an AR operation unit that isbased on the UI information, onto the display 202. The remote controlfunction and the mirroring display function of the smart glasses 101 mayserve as a management function of the application management unit 309,or may serve as one of various applications to be managed by theapplication management unit 309.

At this time, while content of a control target application to becontrolled with the remote control function is being projected anddisplayed on the smart glasses 101, the above-described AR operationunit (mirroring display) that is based on UI information for controllingthe control target application is provided in the field of vision of theuser with the mirroring display function.

In providing mirroring display, the smart glasses 101 according to thepresent exemplary embodiment also implements automatic control ofcreating a free space by slightly reducing a content image of a controltarget application, and providing an AR operation unit that is based onUI information, in the free space. Nevertheless, depending on thesituation, it is also possible to implement the mirroring display bydisplaying a part or all of an image corresponding to the AR operationunit, with being superimposed on the displayed content image of thecontrol target application. For example, in a case where the controltarget application operates in the background, mirroring display may beprovided in a full-screen mode.

The UI information to be used in mirroring display includes screeninformation to be used for projection of an operation unit in the smartwatch 110. The screen information may be one obtained by copying theappearance of the linked device (the smart watch 110). Alternatively,the screen information may be data such as a still image or a movingimage, or may be JavaScript Object Notation (JSON) or Extensible MarkupLanguage (XML) format structured data that enables the smart glasses 101to generate image information.

In some cases, a linked device (the smart watch 110) includes hardware(button or mouse ball) (not illustrated) that is usable in remotecontrol. In such cases, data having been input using the hardware isdelivered to the smart glasses 101 as user instruction information. Atthis time, UI information includes drawing data corresponding to thehardware.

Screen information regarding a screen on the smart watch 110 inmirroring display that is executed under the remote control includesscreen information regarding a control screen such as a “character inputscreen” and/or a “scroll screen” which serve(s) as control informationinput used for an input to a control target application on the smartglasses 101. When an application is ended during the mirroring display,the UI information to be transmitted to the smart glasses 101 includesinformation indicating a mirroring display end. Moreover, when theapplication management unit 328 detects a lapse of a predetermined timefrom a transition of the smart watch 110 into the sleep mode, UIinformation may include information indicating a mirroring display end.

Furthermore, it is also possible for the user to designate, with respectto the smart glasses 101, approval or disapproval of the remote controlfrom a linked device (the smart watch 110) and/or mirroring display ofthe linked device. Such designation is performable by selecting a paireddevice. Alternatively, prompting the user to freely designate variousconditions, such as an ID for identifying a device to be linked, aproduct type, an OS installed on a product, an OS version, anapplication type, and an application version, enables the smart glasses101 to automatically determine approval or disapproval for the remotecontrol from a linked device satisfying the conditions, and/or mirroringdisplay of the linked device. In the present exemplary embodiment, thedescription will be provided of a case where the smart glasses 101uniformly approve the remote control and the mirroring displayirrespective of information regarding the smart watch 110.

FIGS. 4 and 5 are flowcharts illustrating processing, which features thepresent exemplary embodiment, in cooperation between the smart watch 110and the smart glasses 101 that are in a paired state. Processing inwhich the smart watch 110 controls an application installed on the smartglasses 101 will be described. In FIGS. 4 and 5 , it is assumed that thesmart watch 110 and the smart glasses 101 have been subjected to pairingoperation using the pairing unit 327 and the pairing unit 308,respectively.

FIG. 4 is a flowchart illustrating processing to be executed by thesmart watch 110. Referring to this flowchart, a description will beprovided of processing of transmitting information for remote controland/or mirroring display to the smart glasses 101 when the user operatesan application on the smart watch 110. The application management unit328 executes the operation of each step included in this flowchart.

In step S401, the application management unit 328 detect operationinformation about an operation on an application on the smart watch 110,in response to a notification of an input made by the user via theoperation unit 325 and/or an instruction information regarding aninstruction issued from the smart glasses 101 via the communication unit301. Here, the operation information to be detected includes informationregarding an application operation start, an application-operatingstate, or an application operation end. If the application managementunit 328 detects an operation information about an operation on anapparatus (YES in step S401), the processing proceeds to step S402. Instep S402, the application management unit 328 starts the control of anapplication for which a control instruction has been issued, and checkswhether a preset value of ON/OFF of the remote control for theapplication is set to ON in the application-based glasses cooperationinformation (Table 1). In a case where a preset value of ON/OFF of theremote control is set to ON (YES in step S402), the processing proceedsto step S403. In step S403, the application management unit 328 furtherchecks whether a preset value of ON/OFF of mirroring display for thisapplication is set to ON in the application-based glasses cooperationinformation (Table 1). In a case where it is determined in step S403that a preset value of ON/OFF of mirroring display is set to ON (YES instep S403), the processing proceeds to step S404. In step S404, userinstruction information and UI information are transmitted to the smartglasses 101 via the communication unit 322.

In a case where it is determined in step S403 that a preset value ofON/OFF of mirroring display is set to OFF (NO in step S403), theprocessing proceeds to step S405. In step S405, only user instructioninformation is transmitted to the smart glasses 101 via thecommunication unit 322.

In a case where it is determined in step S402 that a preset value ofON/OFF of the remote control is set to OFF (NO in step S402), theprocessing proceeds to step S406. In step S406, the applicationmanagement unit 328 similarly checks whether a preset value of ON/OFF ofthe mirroring display for the application is set to ON in theapplication-based glasses cooperation information (Table 1). In a casewhere it is determined in step S406 that a preset value of ON/OFF ofmirroring display is set to ON (YES in step S406), the processingproceeds to step S407. In step S407, only the UI information istransmitted to the smart glasses 101 via the communication unit 322. Ina case where it is determined in step S406 that a preset value of ON/OFFof mirroring display is set to OFF (NO in step S406), the processing ofthe flowchart illustrated in FIG. 4 is ended.

In a case where no operation information regarding an operation on anapplication is detected in step S401 (NO in step S401), the processingproceeds to step S408. In step S408, the application management unit 328determines whether the smart watch 110 is in the sleep mode. Thefollowing determination as to whether the smart watch 110 is in thesleep mode may be replaced with determination as to whether the smartwatch 110 is in a desired power saving mode to which the state of thesmart watch 110 can transition. In a case where it is determined in stepS408 that the smart watch 110 is not in the sleep mode (NO in stepS408), the processing of the flowchart illustrated in FIG. 4 is ended.

In a case where it is determined in step S408 that the smart watch 110is in the sleep mode (YES in step S408), the processing proceeds to stepS409. In step S409, the application management unit 328 furtherdetermines whether a predetermined time has elapsed since the transitionto the sleep mode. In a case where it is determined in step S409 thatthe predetermined time has not elapsed (NO in step S409), the processingof the flowchart illustrated in FIG. 4 is ended. In a case where it isdetermined in step S409 that the predetermined time has elapsed (YES instep S409), the processing proceeds to step S410. In step S410,information indicating a control end of the remote control is set in theuser instruction information, and information indicating a display endof the mirroring display is set in the UI information. In step S411, theuser instruction information and the UI information are then transmittedto the smart glasses 101.

FIG. 5 is a flowchart illustrating processing, which is to be executedby the smart glasses 101, of receiving control information about theremote control or the mirroring display from the smart watch 110 andexecuting control based on the control information. The applicationmanagement unit 309 executes the operation in the steps included in thisflowchart.

If data is received from a linked device via the communication unit 301,in step S421, the application management unit 309 checks whether thereceived data includes the user instruction information. In a case wherethe received data includes the user instruction information (YES in stepS421), the processing proceeds to step S422. In step S422, theapplication management unit 309 further checks whether the received dataincludes the UI information. In a case where the received data includesthe UI information (YES in step S422), the processing proceeds to stepS423. In step S423, the application management unit 309 starts,continues, or ends the remote control of an application on the smartglasses 101 based on the user instruction information, and furtherstarts, continues, or ends the mirroring display based on the UIinformation.

In a case where it is determined in step S422 that the received datadoes not include the UI information (NO in step S422), the processingproceeds to step S424. In step S424, the application management unit 309checks whether the mirroring display is being provided, based on anexecution status of an application which is being managed. In a casewhere it is determined in step S424 that the mirroring display is beingprovided (YES in step S424), the processing proceeds to step S425. Instep S425, the application management unit 309 determines whether thesmart glasses 101 are in the sleep mode. The following determination asto whether the smart glasses 101 are in the sleep mode may be replacedwith determination as to whether the smart glasses 101 are in a desiredpower saving mode to which the state of the smart glasses 101 cantransition.

In a case where it is determined in step S425 that the smart glasses 101are not in the sleep mode (NO in step S425), the processing proceeds tostep S426. In step S426, the application management unit 309 starts,continues, or ends the remote control based on the user instructioninformation, and continues the mirroring display. In a case where it isdetermined in step S425 that the smart glasses 101 are in the sleep mode(YES in step S425), the processing proceeds to step S427. In step S427,the application management unit 309 further determines whether apredetermined time has elapsed since previous reception of the UIinformation from the linked device (the smart watch 110). In a casewhere the predetermined time has elapsed (YES in step S427), theprocessing proceeds to step S428. In step S428, the applicationmanagement unit 309 ends the mirroring display. In a case where thepredetermined time has not elapsed (NO in step S427), the processingproceeds to step S426. In a case where it is determined in step S424that the mirroring display is not being provided (NO in step S424), theprocessing proceeds to step S429. In step S429, the applicationmanagement unit 309 starts, continues, or ends only the remote controlbased on the user instruction information.

In a case where it is determined in step S421 that the received datafrom the linked device does not include user instruction information (NOin step S421), the processing proceeds to step S430. In step S430, theapplication management unit 309 checks whether the received dataincludes the UI information, as in step S422. In a case where thereceived data includes the UI information (YES in step S430), theprocessing proceeds to step S431. In step S431, the applicationmanagement unit 309 starts, continues, or ends only the mirroringdisplay based on the UI information.

In a case where it is determined in step S430 that the received datadoes not include the UI information (NO in step S430), the processingproceeds to step S432. In step S432, the application management unit 309checks whether the mirroring display is being provided, as in step S424.In a case where it is determined in step S432 that the mirroring displayis being provided (YES in step S432), the processing proceeds to stepS433. In step S433, the application management unit 309 determineswhether the smart glasses 101 are in the sleep mode. In a case where itis determined in step S433 that the smart glasses 101 are not in thesleep mode (NO in step S433), the processing proceeds to step S434. Instep S434, the application management unit 309 continues mirroringdisplay.

In a case where it is determined in step S433 that the smart glasses 101are in the sleep mode (YES in step S433), the processing proceeds tostep S435. In step S435, the application management unit 30) furtherdetermines whether a predetermined time has elapsed since the previousreception of the UI information from the linked device (the smart watch110). In a case where the application management unit 309 determinesthat the predetermined time has elapsed (YES in step S435), theprocessing proceeds to step S436. In step S436, the applicationmanagement unit 309 ends the mirroring display. In a case where theapplication management unit 309 determines that the predetermined timehas not elapsed (NO in step S435), the processing proceeds to step S434.In a case where the application management unit 309 determines in stepS432 that the mirroring display is not being provided (NO in step S432),the processing of the flowchart illustrated in FIG. 5 is ended.

FIGS. 6A to 10C each illustrate display examples on the display 222 ofthe smart watch 110 and the display 202 of the smart glasses 101 thatare provided when a cooperative function of the remote control and/orthe mirroring display is executed. FIGS. 6A to 10C sequentiallyillustrate a series of UI operations including an AR display that arepremised on a use case where the user starts the remote control of thesmart glasses 101 from the smart watch 110, and executes a web searchusing a web browser application on the smart glasses 101.

FIG. 6A illustrates an initial screen 500 of the smart watch 110 and aninitial screen 504 of the smart glasses 101. Application (icons) 501,502, and 503 on the smart watch 110 are a smart watch settingapplication, a web browser application, and a remote control applicationfor controlling an application on the smart glasses 101, respectively.

The display of application icons 505, 506, and 507 are implemented bybeing projected into a field of vision of the user through an ARtechnique of an application executed by the smart glasses 101. Theapplication icons 505, 506, and 507 are icons for an application formaking settings of the smart glasses 101, a web browser application, anda remote controller application having a home electronics remotecontroller function, respectively. Application icons other than theabove-described application icons displayed on the initial screens 500and 504 can also be similarly displayable as an application icon. In acase where the smart glasses 101 recognize a finger of the user, anddetect a selection operation of an icon provided through the ARtechnique, a selected application is activated, and content of theapplication is further displayed in the field of vision of the user.Remote control to be executed by the smart watch 110 will be describedbelow. The remote control is performable in combination with anoperation to be executed in response to the recognition of an input madeby the user on the smart glasses 101.

If the user presses the remote control application (icon) 503 on theinitial screen 500 illustrated in FIG. 6A, a remote control settingscreen 508 illustrated in FIG. 6B is displayed. The remote controlsetting screen 508 includes a control enabling button 509 for enablingand starting the remote control, and a control target applicationselection button 510 for transition to a control target applicationselection screen to be displayed when an application on the smartglasses 101 is controlled. The control enabling button 509 is set to OFFin an initial state. By pressing the control enabling button 509 asillustrated in FIG. 6B, a toggle button of ON/OFF is slid and thecontrol enabling button 509 is switched to control ON.

When the control enabling button 509 is switched to control ON, in acase where pairing of the smart watch 110 and the smart glasses 101 hasnot been executed, a popup screen 511 for inexecution of the pairing isdisplayed as illustrated in FIG. 6C. In the popup screen 511 forinexecution of the pairing, a warning message indicating that pairinghas not been executed yet and a candidate list 512 of target smartglasses 101 to which the smart watch 110 is able to be paired isdisplayed. In the case example of the popup screen 511 for inexecutionof the pairing, the candidate list 512 indicates that pairing to thesmart glasses 101 named “Glasses-1” and “Glasses-2” is executable overnear field communication. If the user selects “Glasses-1”, anauthentication screen (not illustrated) for pairing is displayed, andthe user inputs an authentication ID and/or an authentication password,so that authentication execution and pairing execution are completed. Toclose the popup screen 511 for inexecution of the pairing, a closebutton 513 is pressed.

In a case where pairing has been executed when the control is set to ONusing the control enabling button 509 as illustrated in FIG. 6B, or in acase where pairing execution has been completed based on the popupscreen 511 for inexecution of the pairing illustrated in FIG. 6C, theremote control is started.

FIG. 7A illustrates a UI display state in which a control targetapplication selection screen 514 for controlling an application on thesmart glasses 101 that is to be controlled is displayed in aremotely-controlling state. The screen transitions to this state by theuser pressing the above-described control target application selectionbutton 510 illustrated in FIG. 6B. A setting control application (icon)515, a web browser control application (icon) 516, and a remotecontroller control application (icon) 517 that are displayed on thecontrol target application selection screen 514 are icons used forrespectively controlling applications on the smart glasses 101.

If a selection operation of the web browser control application 516 thatis performed by the user on the smart watch 110 is detected, a webbrowser control screen 519 illustrated in FIG. 7B is displayed on thesmart watch 110. In addition, a web browser screen 528 which is a screendisplayed after the activation of the web browser application 506 to becontrolled, and a control display 527 that is displayed during thecontrol and indicates that the remote control is being executed aredisplayed on the smart glasses 101. As described above, the display onthe smart glasses 101 is a projection display in the field of vision ofthe user through the AR technique.

In the display 527 which is displayed during the control, a name of alinked terminal that instructs the remote control is displayed in such amanner that a linked terminal controlling the smart glasses 101 isrecognizable. In this example, that the smart glasses 101 are beingcontrolled by the smart watch 110 named “Watch-1” is displayed. The webbrowser screen 528 is a search screen. The web browser screen 528includes a return button 529 for returning to a screen displayed beforesearch, after search is executed on the search screen, an input field530 for entering a search word, and a click button 531 for executing asearch based on a word entered into the input field 530. The web browserscreen 528 further includes a pointer 532 for designating a region onthe web browser screen 528 while moving on the web browser screen 528, abrowser update button, a button indicating a minimized or maximizeddisplay of the browser, a button for closing a browser, and a URLdisplay field.

A return button 520 on the web browser control screen 519 on the smartwatch 110 corresponds to the above-described return button 529, and isan instruction button for pressing the return button 529. In addition, aBACK button 524 on the web browser control screen 519 is a button forending the web browser control application, and returning to the controltarget application selection screen 514, which is a previous screen. Ifthe user presses an END button 525, the web browser control application516 and the remote control application 503 are ended, the web browsercontrol screen 519 is closed, and the initial screen 500 is displayed.

The web browser control screen 519 is an instruction screen for issuinga control instruction of the web browser application 506 on the smartglasses 101, but the web browser control screen 519 further includesapplication icons for instruction control applications each having afunction of issuing a specific operation instruction.

Examples of instruction control applications include a pointer movementinstruction control application 521, a keyboard input instructioncontrol application 522, and a touch pad instruction control application523. With the pointer movement instruction control application 521, aninstruction to move the pointer 532 in a direction indicated by an arrowkey is issued. With the keyboard input instruction control application522, a character input instruction is issued. With the touch padinstruction control application 523, an instruction equivalent to thatwith a touch pad function included in an existing laptop personalcomputer (PC) is issuable. In addition, the control instructionapplications also include a scroll application for issuing a scrollinstruction, a pinch-in/out application for issuing a pinch-in/outinstruction, a flick application for sliding a screen upward, downward,leftward, or rightward by a flick, and a tap application specialized ina tap instruction.

Regarding the applications on the smart watch 110, which have beendescribed in conjunction with FIGS. 6A to 6C, 7A, and 7B, Table 2specifically indicates application-based glasses cooperation informationin Table 1 described above.

TABLE 2 Application-Based Glasses Cooperation Information ApplicationApplication Remote Mirroring ID Name Control Display 0004 Remote Control503 ON OFF 0005 Setting Control 515 ON OFF 0006 Web Browser Control 516ON OFF 0007 Remote Controller Control 517 ON ON 0008 Pointer MovementInstruction ON ON Control 521 0009 Keyboard Input Instruction ON ONControl 522 0010 Touch Pad Instruction ON OFF Control 523

The start of the above-described remote control illustrated in FIGS. 6Ato 6C is executed based on the remote control ON/OFF setting for theremote control application 503 (row with an application ID: 0004) beingset to ON in Table 2. Here, the mirroring display is set to OFF, themirroring display is not executed at the time point. The mirroringdisplay may be preset to ON when the remote control application 503 isactivated. In such a case, the remote control setting screen 508 ismirrored and displayed on the smart glasses 101 at a start timing of theremote control illustrated in FIGS. 6A to 6C.

After the web browser control application 516 is pressed on the controltarget application selection screen 514 illustrated in FIG. 7A, asdescribed above, screen display as illustrated in FIG. 7B is provided.At this time, while the remote control is being executed, the mirroringdisplay is not provided. This state is based on the remote controlON/OFF setting and the mirroring display ON/OFF setting for the webbrowser control application 516 (row with an application ID: 0006) beingpreset to ON and OFF, respectively, in Table 2. In a case where themirroring display ON/OFF setting for the web browser control applicationis preset to ON, the web browser control screen 519 is mirrored anddisplayed on the smart glasses 101 at the timing of FIG. 7B.

FIGS. 8A and 8B illustrate examples of display to which the displaytransitions in a case where the user selects the pointer movementinstruction control application 521 on the web browser control screen519 illustrated in FIG. 7B. The displays in FIGS. 8A and 8B are providedwhen a control application that performs mirroring display is executed.FIG. 8A illustrates an example of display to be provided in a case wherethe mirroring display is started.

The remote control ON/OFF setting and the mirroring display ON/OFFsetting for the pointer movement instruction control application (rowwith an application ID: 0008) are both preset to ON in Table 2. Theremote control is continued and the mirroring display is started basedon the preset values.

A pointer movement instruction control screen 533 is displayed on thesmart watch 110, an arrow key 534 enables instructions to move thepointer 532 upward, downward, leftward, and rightward. Pressing of a TAPbutton at the center of the arrow key 534 enables a tap instruction. ABACK button 535 is used for ending the pointer movement instructioncontrol application 521 and returning to the web browser control screen519, which is a previous screen. If the user presses the END button 525,the pointer movement instruction control application 521, the webbrowser control application 516, and the remote control application 503are ended, the screen is closed, and the initial screen 500 isdisplayed.

In FIG. 8A, a mirroring display message 536 indicating that mirroringdisplay is being provided is additionally displayed in the field ofvision of the user through the smart glasses 101. The mirroring displaymessage 536 includes a name of a linked terminal of which a UI ismirrored and displayed so that the linked terminal is identifiable. Inthis example, the mirroring display message 536 indicates that a UI ofthe smart watch 110 named “Watch-1” is displayed. A mirroring displayimage 537 mirrors and displays a UI of the pointer movement instructioncontrol screen 533 that is being displayed at this timing on the smartwatch 110 named “Watch-1”. In addition, the web browser screen 528illustrated in FIG. 8A is moved in display position and reduced inscreen size from the web browser screen 528 illustrated in FIG. 7B thathad been displayed before the mirroring display. In this example, suchan automatic control is added to ensure a region for displaying theabove-described mirroring display image 537.

In a case where the user has operated a downward arrow of the arrow key534 on the smart watch 110 as illustrated in FIG. 8A, the operation isreflected in the mirroring display image 537, and the pointer 532 on theweb browser screen 528 moves downward in accordance with theinstruction. The operation received on the smart watch 110 is reflectedin the mirroring display image 537, thus enabling the user to re-confirmthe operation performed by himself/herself, within the field of vision.This prevents a redundant operation and an erroneous operation. FIG. 8Aillustrates a state in which the pointer 532 has moved to a positiononto the input field 530.

Subsequently, if the user presses the TAP button provided at the centerof the arrow key 534, as illustrated in FIG. 8B, the operation issimilarly reflected in the mirroring display image 537, and the inputfield 530 is selected because the pointer 532 exists on the input field530.

If the user presses the BACK button 535 on the pointer movementinstruction control screen 533 illustrated in FIG. 8A or 8B, the screendisplayed when the BACK button 535 is pressed is similarly reflected inthe mirroring display image 537, and the screen returns to the webbrowser control screen 519 illustrated in FIG. 7B. The pointer movementinstruction control application is then ended, and the web browsercontrol application 516 enters an operating state. Thus, in accordancewith information regarding the web browser control application 516 withthe application ID “0006” in Table 2, the mirroring display is ended(set to OFF).

For a simple operation unit as illustrated in the mirroring displayimage 537 in FIGS. 8A and 8B, it may be possible for the user to inputan operation into the smart watch 110 while seeing the mirroring displayimage 537, without visually checking the display on the smart watch 110.Thus, it is possible for the user to control an AR application only byperforming a slight operation on his/her wrist without looking away,such as facing down toward the direction of the smart watch 110.

FIGS. 9A and 9B each illustrate examples of display to which the displaytransitions in a case where the user selects the keyboard inputinstruction control application 522 on the web browser control screen519 illustrated in FIG. 7B. The display in each of FIGS. 9A and 9B isdifferent from the examples illustrated in FIGS. 8A and 8B that areprovided when a control application for performing the mirroring displayis executed. The mirroring display is started as illustrated in FIG. 9A.A keyboard input instruction control screen 538 is displayed on thesmart watch 110. The keyboard input instruction control screen 538A thendisplays a keyboard 539 and a BACK button 540 for ending the keyboardinput instruction control application 522 and returning to the webbrowser control screen 519, which is a previous screen. At this time,the keyboard input instruction control application 522 enters anoperating state, so that the mirroring display is restarted (set to ON),in accordance with information regarding the keyboard input instructioncontrol application with an application ID “0008” in Table 2, and amirroring display image 541 is mirrored and displayed. In the mirroringdisplay image 541, a UI of the keyboard input instruction control screen538 displayed at this timing on the smart watch 110 named “Watch-1” ismirrored and displayed.

If the user selects a character “j”, for example, on the keyboard 539illustrated in FIG. 9B, input candidate words starting with “j” aredisplayed in a candidate word list 542. An OK button 543 for confirmingselection when the user selects a word from the candidate word list 542is also displayed. Subsequently, the UI of the keyboard inputinstruction control screen 538 at this timing is reflected in themirroring display image 541, and the selected character “j” is alsodisplayed in the input field 530 on the web browser screen 528.

For example, if the user selects “Japan” from among candidate words inthe candidate word list 542 illustrated in FIG. 9B, the selection isreflected in the mirroring display image 541, and “Japan” is displayedin the input field 530 on the web browser screen 528 in acandidate-selected state. If the user presses the OK button 543 in thisstate, “Japan” is similarly reflected in the mirroring display image541, and “Japan” in the input field 530 on the web browser screen 528 isdisplayed in a confirmed state. Thus, the web browser screen 528 entersa next character input standby state.

If the user presses a BACK button 540 on the keyboard input instructioncontrol screen 538 illustrated in FIG. 9A or 9B, the screen returns tothe web browser control screen 519 illustrated in FIG. 7B. The keyboardinput instruction control application 522 ends at this timing, and theweb browser control application 516 enters an operating state. Thus, inaccordance with information regarding the web browser controlapplication 516 with the application ID “0006” in Table 2, the mirroringdisplay is ended (set to OFF).

FIGS. 10A to 10C each illustrate an example of a display to which thedisplay transitions in a case where the user selects the touch padinstruction control application (icon) 523 on the web browser controlscreen 519 illustrated in FIG. 7B. Each display is provided when acontrol application that does not perform the mirroring display isexecuted. An application on the web browser screen 528 is controlledwithout the mirroring display, as illustrated in FIG. 10A. A touch padinstruction control screen 544 is displayed on the smart watch 110. Thetouch pad instruction control screen 544 displays thereon a touch padregion 545 for issuing a touch pad instruction, and a BACK button 546for ending the touch pad instruction control application and returningto the web browser control screen 519, which is a previous screen. Inaccordance with information regarding the touch pad instruction controlapplication 523 with an application ID “0010” in Table 2, the touch padinstruction control application continues the remote control with themirroring display set to OFF. Thus, when the state transitions from thestate illustrated in FIG. 7B, to the state illustrated in FIG. 10A, thedisplay on the smart glasses 101 remains unchanged. If the user moveshis/her finger in the touch pad region 545 in FIG. 10A in such a manneras to slide the finger rightward, the pointer 532 on the web browserscreen 528 also moves rightward in conjunction with the movement. FIG.10A illustrates a state in which the pointer 532 has moved to a positionon a Click button 531.

Subsequently, if the user taps his/her finger twice in the touch padregion 545 illustrated in FIG. 10B, the Click button 531 designated bythe pointer 532 on the web browser screen 528 is clicked in conjunctionwith the movement. Web search is accordingly executed based on the word“Japan” entered in the input field 530. FIG. 10C illustrates a screen tobe displayed after the search operation, and a search result isdisplayed as in the web browser screen 528. The number of search results(hits), link information regarding a link to a hit article, and imageinformation are displayed in a list in a search result informationregion 547 on the web browser screen 528.

If the user presses a BACK button 546 on the touch pad instructioncontrol screen 544 illustrated in FIGS. 10A to 10C, the screen returnsto the web browser control screen 519 illustrated in FIG. 7B.

The above-described UI operations illustrated in FIGS. 6A to 10C enablethe setting of the mirroring display to be set to ON/OFF depending on anoperated application when an application on the smart glasses 101 isremotely controlled from the smart watch 110.

In a series of UI operations as illustrated in FIGS. 6A to 10C, thesmart glasses 101 are capable of receiving an instruction issued by theuser via the operation unit 304, also when the smart glasses 101 areremotely controlled from the smart watch 110, as described above. Thus,while a UI of the smart watch 110 is being mirrored and displayed, anoperation performed by the user via the operation unit 304 of the smartglasses 101 sometimes interrupts. In this case, the smart glasses 101once end the mirroring display. In this case, if the smart watch 110redetects an operation of an application for which the mirroring displayON/OFF setting is set to ON in Table 2, and issues a mirroringinstruction to the smart glasses 101, the smart glasses 101 restartmirroring start.

In the use case of the UI operations illustrated in FIGS. 6A to 10C, themirroring display is started by the user selecting the control targetapplication selection button 510 in FIG. 6B, and then pressing the webbrowser control application 516 in FIG. 7A. In other words, this is ause case where the application management unit 328 of the smart watch110 detects operation information regarding an operation on the webbrowser control application 516 on the smart watch 110 in response toreceiving a notification of an “input made by the user via the operationunit 325” (step S401).

In contrast, as a different use case, there is also a use case where theapplication management unit 328 of the smart watch 110 activates acontrol instruction application on the smart watch 110 in response toreceiving a control instruction application activation request from thesmart glasses 101 via the communication unit 301. At the time, the smartwatch 110 that has received a notification activates a controlinstruction application, and detects the activation of the controlinstruction application (step S401). The different use case will bebriefly described with reference to FIGS. 6A to 6C.

First of all, in the case of this use case, the control targetapplication selection button 510 in FIG. 6B is not selected, a Closebutton (not illustrated) on the remote control setting screen 508 ispressed while the remote control in the background is being executed andthe initial screen 500 remains displayed.

Subsequently, in a standby state on the initial screen 500, for example,the user selects the input field 530 on the web browser screen 528displayed on the smart glasses 101, using the operation unit 304 of thesmart glasses 101. The application management unit 309 then detects theselection of the input field 530, and the application management unit309 requests the smart watch 110 to activate the keyboard inputinstruction control application 522 illustrated in FIGS. 6A to 6C, viathe communication unit 301.

Subsequently, if the smart watch 110 receives the request for activationof the keyboard input instruction control application 522, the smartwatch 110 activates the keyboard input instruction control application522, and displays the keyboard input instruction control screen 538. Theuser makes an input using the displayed keyboard input instructioncontrol screen 538.

After that, if the user ends the character input and presses a Closebutton (not illustrated) on the keyboard input instruction controlscreen 538, the smart watch 110 ends the keyboard input instructioncontrol application 522, and the smart watch 110 returns to the standbystate on the initial screen 500 again.

According to the above-described exemplary embodiment, it is possible toenhance the usability in cooperation executed when a function of anapplication on smart glasses is remotely controlled by a smart watch,using the AR technique.

In the present exemplary embodiment, the description has been providedusing the term “mirroring display”, but a display image (e.g., themirroring display image 537 in FIG. 8A) exactly matching an operationunit or operability displayed on the smart watch 110 is not necessarilyto be projected by the smart glasses 101. It is sufficient that virtualdisplay of an operation unit is realized to such a degree that theoperability is recognizable, and the mirroring display may be realizedby any image indicating an operation unit customized by the user, anapplication provider, or the smart glasses 101.

Second Exemplary Embodiment

The user uses the smart watch 110 and the smart glasses 101 incombination in various usage environments. Thus, in consideration of ausage environment and an environment change during cooperation betweenthese devices, it is possible to further enhance the usability for theuser at the time of the cooperation.

In the second exemplary embodiment, an exemplary embodiment in which ausage environment and an environment change during cooperation areconsidered will be described based on a difference from the firstexemplary embodiment.

In the software module configuration (FIG. 3B) of the smart watch 110according to the present exemplary embodiment, the applicationmanagement unit 328 is capable of presetting, as application-basedglasses cooperation information, a function to be enabled in each usageenvironment during cooperation with the smart glasses 101. Theapplication management unit 328 is capable of presetting also a functionto be activated when a change in usage environment of the user isdetected. These settings are made for each of various applications onthe smart watch 110. As an example, settings are predetermined asindicated in Table 3.

TABLE 3 Application-based glasses cooperation information Remote ControlMirroring Display Usage Usage Watch Screen Lock Application ApplicationON/OFF environment ON/OFF environment ON/OFF Activation ID Name settingsetting setting setting setting environment 0001 Application0001 ON“Crowded ON Crowded ON Crowded train or train train walking” 0002Application0002 ON — ON Walking ON Walking 0003 Application0003 OFF —OFF While ON — directly viewing linked screen

In glasses cooperation that is based on the application-based glassescooperation information in Table 3, in which usage environment theremote control and the mirroring display described in the firstexemplary embodiment are enabled is predetermined. In the secondexemplary embodiment, not only the “ON/OFF setting” for setting whetherto enable or disable a function, but also a usage environment in whichthe function is to be enabled (set to ON) is set as a “usage environmentsetting”.

For example, an application with an application ID “0001” in Table 3will be described as an example. First of all, the ON/OFF setting ofremote control is set to “ON”, and a usage environment setting is set to“crowded train or walking”. In this case, if it is detected that a userenvironment is “crowded train or walking”, the remote control functionis started. After that, if it is detected that a user environment isneither “crowded train” nor “walking”, the remote control function isended. The ON/OFF setting of the mirroring display for the applicationwith an application ID “0001” is also set to “ON”, and a usageenvironment setting is set to “crowded train”. In this case, if it isdetected that a user environment is “crowded train”, the mirroringdisplay provided by the mirroring display function is started. Inaddition, if it is detected that a user environment is not “crowdedtrain”, the mirroring display provided by the mirroring display functionis ended.

The application management unit 309 executes an environment changedetection as to whether a user environment is “crowded train” or“walking”, based on information from the sensor unit 209 or the camera210 included in the smart glasses 101. The application management unit309 also manages an environmental status indicating a current userenvironment, in the ROM 203 or the RAM 204 via the storage unit 302. Ifthe smart glasses 101 detect an environment change using the applicationmanagement unit 309, the smart glasses 101 notify the smart watch 110 ofthe environment change detection. If the smart watch 110 receives theenvironment change detection, the smart watch 110 checks an applicationcurrently being operated by the smart watch 110. Then, in a case wherethe smart watch 110 is operating any application, the “remote control”ON/OFF setting and the “mirroring display” ON/OFF setting are set to ONor OFF, based on the application-based glasses cooperation informationin Table 3.

There are applications for which “- (hyphen)” is designated in usageenvironment settings in Table 3. This indicates that a usage environmentis undesignated. In this case, a function is enabled or disabled inaccordance with the “ON/OFF setting” in all environments. There is anapplication for which “while directly viewing a linked screen” isdesignated in a usage environment setting in Table 3. This indicates astate in which the user is physically viewing the screen (the display222) of the linked smart watch 110 through a lens of the smart glasses101. As for an application with an application ID “0003” in Table 3, theON/OFF setting of mirroring display is set to OFF, and a usageenvironment setting is set to “while directly viewing a linked screen”.In this case, if the user directly views the screen of the smart watch110 during the mirroring display on the smart glasses 101, the mirroringdisplay is ended. Moreover, if the user looks away from the screen ofthe smart watch 110, the mirroring display is restarted. With thisconfiguration, the user is freed from vexatiousness due to duplicatedisplay of the same screen when, for example, the same screen is alsomirrored and displayed on the smart glasses 101 while the user isdirectly viewing the screen of the smart watch 110.

Furthermore, it has been described that, in a case where “crowded trainor walking” is set in a usage environment setting as described above,this setting indicates “crowded train” or “walking”. As a similarsetting, if “while directly viewing a linked screen and walking” is setin a usage environment setting, which is not described in the caseexample in Table 3, this setting indicates “while directly viewing alinked screen” and “walking”.

As described above, the definition of an “environment” in this secondexemplary embodiment is not limited to an environment itself surroundingthe user, such as “crowded train”, and an environment including a userstate such as “walking” or “while directly viewing a linked screen” isdefined as an “environment,” as indicated in the usage environmentsettings in Table 3. Table 3 provides an example in which values such as“crowded train” and “walking” are managed in the “usage environmentsetting” for the sake of explanatory convenience. In practice, settingvalue IDs allocated to the respective environments may be managed inTable 3.

Furthermore, in the second exemplary embodiment, a watch screen lockfunction of locking the screen of the smart watch 110 in a case where aspecific environment change is detected is also provided. This functionis specified on the columns of “ON/OFF setting” and “activationenvironment” on a “watch screen lock” column in Table 3. For example, asfor an application with an application ID “0001” in Table 3, the ON/OFFsetting on the “watch screen lock” column is set to “ON”, and theactivation environment is set to “crowded train”. In this case, it isindicated that, if the transition of a user environment from an“environment other than a crowded train environment” to the “crowdedtrain environment” is detected, the watch screen lock function isexecuted. The watch screen lock function is used for the purpose ofreducing the possibility that another user touches the screen (thedisplay unit 111) of the smart watch 110 in a crowded train state, anderroneous control is executed in the remote control on the smart glasses101. If the screen of the smart watch 110 is once locked when the userenters the crowded train state, erroneous control caused by touch fromanother person can be prevented.

After the screen is locked by the watch screen lock function, promptingthe user to log into the smart watch 110 again by, for example,fingerprint authentication using the screen ensures a screen operationin a state in which the authenticated user touches the screen.

As another intended purpose of the watch screen lock function, lockingthe screen of the smart watch 110 when the state of the user transitionsto “walking” is also effective. When the user remotely controls thesmart glasses 101 using the screen of the smart watch 110, the usersometimes directly views the screen of the smart watch 110. If the userstarts walking in this state, the user walks while directly viewing thescreen of the smart watch 110, and cannot notice a dangerous environmentpossibly existing around the user. Thus, by once locking the screen whenthe state of the user transitions to “walking”, the user's attention tothe screen of the smart watch 110 can be once distracted, and anopportunity to give an eye to a surrounding environment can be provided.When the screen is locked by the watch screen lock function at thistime, a warning may be displayed on the screen.

In addition, when the user unlocks the screen lock again while the usercontinues to be in a “walking” state, a guidance to execute themirroring display may be provided. At the time, a series of processes ofdisplaying a warning when the screen is locked, and executing themirroring display after the screen lock is unlocked may be included inthe “watch screen lock” function.

Furthermore, as a method of unlocking the screen lock executed by the“watch screen lock” function, a “watch screen unlock” function notdescribed in Table 3 may be provided as one of cooperative functions.For example, a method of designating an environment indicating “stopped(walking OFF)” in the usage environment setting in Table 3, and settingthe ON/OFF setting of “watch screen unlock” to ON at the time isconceivable.

FIGS. 11 to 13 are flowcharts illustrating processing featuring thesecond exemplary embodiment. Each flowchart is executed in the smartwatch 110 and the smart glasses 101 in a pairing state when the smartwatch 110 controls an application installed on the smart glasses 101. InFIGS. 11 to 13 , the smart watch 110 and the smart glasses 101 areassumed to have been subjected to pairing using the pairing unit 327 andthe pairing unit 308, respectively.

FIG. 11 is a flowchart illustrating processing of transmittingenvironment change detection to the smart watch 110 when a userenvironment change is detected in the smart glasses 101. The applicationmanagement unit 309 executes the operations in the steps included inthis flowchart.

In step S601, the application management unit 309 detects whether theenvironment surrounding the user has been changed based on informationfrom the sensor unit 209 or the camera 210. For example, using thesensor unit 209, the application management unit 309 detects whether theuser is walking, based on information from the GPS or the accelerationsensor. In addition, the application management unit 309 detects whetherthe user is in the crowded train state, based on information from theproximity sensor included in the sensor unit 209 or image informationfrom the camera 210. Furthermore, in a case where the camera 210includes a function of capturing an image of an object in almost thesame direction as an eye direction of the user of the smart glasses 101,the application management unit 309 detects whether the user is directlyviewing the screen of the linked device (the smart watch 110) that is ina cooperation state, based on image information from the camera 210. Theabove-described detection method is an example, and the applicationmanagement unit 309 detects an environment change based on a functionincluded in the smart glasses 101.

In a case where it is determined in step S601 that an environment changehas not been detected (NO in step S601), the application management unit309 ends the processing of the flowchart illustrated in FIG. 11 . In acase where an environment change has been detected (YES in step S601),the processing proceeds to step S602. In step S602, the applicationmanagement unit 309 stores the detected environment information aslatest environment information into the RAM 204 via the storage unit302. Then, in step S603, the application management unit 309 notifiesthe smart watch 110 of the detected environment information.

The latest environment information is stored into a table as illustratedin Table 4.

TABLE 4 Latest Environment Information Environment State Walking ONCrowded Train OFF While Directly OFF Viewing Linked Screen

The application management unit 309 presets environment items as listedon an “environment” column in Table 4, which are detectable by theapplication management unit 309. Then, for example, in a case where anenvironment changes from a stopped state to a walking state, the“walking” state is updated to ON. After that, in a case where anenvironment changes from the walking state to the stopped state, the“walking” state is updated to OFF.

FIG. 12 is a flowchart illustrating processing of transmitting, when thesmart watch 110 receives the environment information from the smartglasses 101, control information regarding the remote control and/or themirroring display that has/have been determined based on the receivedenvironment information, to the smart glasses 101. The applicationmanagement unit 328 executes the operations in the steps included inthis flowchart.

First of all, in step S611, the application management unit 328determines whether the environment information has been received fromthe smart glasses 101. In a case where environment information has beenreceived (YES in step S611), the processing proceeds to step S612. In acase where environment information has not been received (NO in stepS611), the processing proceeds to step S613. In step S612, theapplication management unit 328 stores the received latest environmentinformation (Table 4) into the RAM 224 via the storage unit 323. Theoperation in step S613 is equivalent to that in step S401. If theapplication management unit 328 detects an operation information aboutan operation on an apparatus (YES in step S613), the processing proceedsto step S614. If the application management unit 328 detects nooperation information about an operation on an apparatus (NO in stepS613), the processing proceeds to step S621.

The operations in step S621 and step S622 are respectively equivalent tothe operations in step S408 and step S409. The operation in step S623 isequivalent to an operation obtained by combining the both operations instep S410 and step S411. In step S623, in a case where cooperativefunctions that are to be executed on the smart glasses 101, other thanthe remote control function and the mirroring display function, areprovided in the above-described application-based glasses cooperationinformation (Table 3), all of these cooperative functions are ended aswell. The smart glasses 101 are also notified of the end result.

In step S614, the application management unit 328 acquires the latestenvironment information (Table 4) stored in step S612. In step S615, theapplication management unit 328 determines whether a “cooperativefunction to be executed on the smart watch 110” that satisfies a presetvalue in the usage environment setting in Table 3 is included based onan environment of which the state is set to ON in Table 4. If the“cooperative function to be executed on the smart watch 110” is included(YES in step S615), the processing proceeds to step S616. In step S616,the cooperative function is executed by the smart watch 110″. Here, the“cooperative function to be executed on the smart watch 110” is the“watch screen lock” function in the example in Table 3. The watch screenlock function is a function of locking the screen of the smart watch110, so that the watch screen lock function is to be executed on thesmart watch 110. As described above in conjunction with Table 3, the“watch screen lock” function may include a function of displaying awarning in the locked state and a function of executing mirroringdisplay after the screen lock is unlocked. Moreover, as similarlydescribed above in conjunction with Table 3, the “watch screen unlock”function is also included in the “cooperative function to be executed onthe smart watch 110”.

Subsequently, in step S617, the application management unit 328determines whether a “cooperative function to be executed on the smartglasses 101” that satisfies a preset value in the usage environmentsetting in Table 3 is included based on an environment of which thestate is set to ON in Table 4. If the “cooperative function to beexecuted on the smart glasses 101” is included (YES in step S617), theprocessing proceeds to step S618. In step S618, the applicationmanagement unit 328 notifies the smart glasses 101 of state informationabout the cooperative function. Here, the “cooperative function to beexecuted on the smart glasses 101” includes the “remote control”function and the “mirroring display” function in the example in Table 3.The remote control function controls an application on the smart glasses101. The mirroring display function performs mirroring display on thesmart glasses 101. Thus, these functions are to be executed on the smartglasses 101. The state information about the cooperative function ofwhich the smart glasses 101 are notified in step S618 includesdetermination results obtained by determining whether remote control andmirroring display are set to ON or OFF, based on Table 4, for example.When the notification is made, information regarding an application IDand an application name of an application of which operation informationhas been detected in step S613 is also included in the stateinformation.

Heretofore, the functions to be executed on the smart watch 110 and thefunctions to be executed on the smart glasses 101 that are set in Tables3 and 4 have been described. Nevertheless, information for determining afunction into which a target cooperative function falls may be madesettable by further adding an attribute to the application-based glassescooperation information (Table 3), or may be managed in another table.

In step S619, the application management unit 328 determines whether afunction that is being performed in cooperation with the smart glasses101 is included. In the determination, if it is determined that a“cooperative function to be executed on the smart glasses 101” thatsatisfies a preset value in the usage environment setting in Table 3 isincluded, based on an environment of which the state is set to ON inTable 4, as in step S617, and it is further determined that a functionto be enabled (set to ON) is included, it is determined that a functionbeing in cooperation is included. In addition to such a determinationmethod, a past state information notification history in step S618 maybe stored in the storage unit 323, and it may be determined whether afunction that is being performed in cooperation is included, based onthe past notification history. If it is determined in step S619 that afunction that is being performed in cooperation with the smart glasses101 is included (YES in step S619), the processing proceeds to stepS620. If it is determined in step S619 that a function that is beingperformed in cooperation with the smart glasses 101 is not included (NOin step S619), the processing in the flowchart illustrated in FIG. 12 isended.

The operation in step S620 is equivalent to those in step S404, S405, orS407 of FIG. 4 . For example, if the remote control function is incooperation, user instruction information is transmitted as applicationoperation information. If the mirroring display function is incooperation, the UI information is transmitted as application operationinformation.

FIG. 13 is a flowchart illustrating processing to be executed when thesmart glasses 101 receive, from the smart watch 110, “state informationfor setting ON/OFF a cooperative function” or “application operationinformation corresponding to an in-cooperation cooperative function”.The application management unit 309 executes processing steps includedin this flowchart.

In step S631, the application management unit 309 determines whether“state information about a cooperative function” has been received froma linked device (the smart watch 110). In a case where the “stateinformation about a cooperative function” has been received (YES in stepS631), the processing proceeds to step S632. In a case where “stateinformation about a cooperative function” has not been received (NO instep S631), the processing proceeds to step S633. The “state informationabout a cooperative function” to be received in step S631 is informationincluded in the notification from the smart watch 110 transmitted instep S618 or S623 of FIG. 12 .

In step S632, the received state information is reflected as processingto be executed on the smart glasses 101. More specifically, in a casewhere the state information indicating the remote control ON has beenreceived, the control of an application on the smart glasses 101 that iscontrollable by an application on the smart watch 110 that has anapplication ID included in the state information transmitted in stepS618 is started. In a case where the state information indicating theremote control OFF has been received, the control of the application isended. In a case where state information indicating the mirroringdisplay ON has been received, the application management unit 309activates a mirroring display application on the smart glasses 101, andthe mirroring display is accordingly started. In a case where the stateinformation indicating the mirroring display OFF has been received, themirroring display is ended.

In step S633, the application management unit 309 determines whether“application operation information corresponding to an in-cooperationcooperative function” has been received from the smart watch 110. In acase where “application operation information corresponding to anin-cooperation cooperative function” has been received (YES in stepS633), the processing proceeds to step S634. In a case where“application operation information corresponding to an in-cooperationcooperative function” has not been received (NO in step S633), theprocessing proceeds to step S635. The “application operation informationcorresponding to an in-cooperation cooperative function” is informationof which the smart glasses 101 are notified by the smart watch 110 instep S620 of FIG. 12 .

In step S634, the received application operation information isreflected as processing to be executed on the smart glasses 101. Morespecifically, in a case where the in-cooperation cooperative function isthe remote control function, control that is based on user instructioninformation serving as the application operation information isreflected in an application on the smart glasses 101. In a case wherethe in-cooperation cooperative function is the mirroring displayfunction, the application operation information is the UI informationcorresponding to information regarding display content to be mirroredand displayed on the smart glasses 101. In this example, for example,display content to be changed by the user operating an application onthe smart watch 110 is the UI information, and the mirroring display isupdated based on the UI information.

The operations in steps S635 to S639 are equivalent to those in stepsS424 to S428, and thus, descriptions thereof will be omitted.

According to the above-described flowcharts, it is possible to enhancethe usability reflecting a usage environment of the user, in thecooperation that is executed when a smart glass function is remotelycontrolled by a smart watch.

In the second exemplary embodiment, the smart glasses 101 performenvironment change detection and notify the smart watch 110 of a resultof the environment change detection (FIG. 11 ). The environment changedetection may be executed by another configuration. For example, thesmart watch 110 may execute the environment change detection using theapplication management unit 328 based on information from the sensorunit 229 included in the smart watch 110. In addition, environmentmodes, such as a normal mode, a crowded train mode, and a walking mode,may be prepared in the smart glasses 101 or the smart watch 110, and theuser may be enabled to select any environment mode from among theseenvironment modes. Then, the mode switching of an environment mode thatis performed by the user may be used as an alternate method ofenvironment change detection.

In the above-described first and second exemplary embodiments, theconfiguration of each wearable terminal, the configurations of softwaremodules, and flowcharts regarding application notification of a wearableterminal have been described. These are only examples, and the presentinvention is not limited to these.

According to each of the above-described exemplary embodiments, and thepresent invention including a combination of these, a high-performanceoperation unit for dealing with various operations to be used forvarious applications is not necessarily to be provide in smart glasses.This eliminates design restrictions imposed to mount a number of suchoperation units in smart glasses. In other words, it can be said thatthe present invention contributes largely not only to enhancement inusability but also to design improvement of the smart glasses, and mightalso contribute to weight saving.

Other Exemplary Embodiments

The present invention also includes an apparatus or a system obtained byappropriately combining the above-described exemplary embodiments, and amethod for the same.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

The present invention is not limited to the above-described exemplaryembodiments, and various changes and modifications can be made withoutdeparting from the spirit and the scope of the present invention. Thus,the following claims are appended to publicize the scope of the presentinvention.

According to an exemplary embodiment of the present invention, it ispossible to enhance usability in executing cooperation between aglasses-type information device and a different wearable terminal (smartwatch, etc.) and remotely controlling the glasses-type informationdevice.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

1. A glasses-type information device configured to execute anapplication, the glasses-type information device comprising: at leastone memory storing instructions; and at least one processor executingthe instructions causing the glasses-type information device to: executedisplay control in such a manner that an image corresponding to anoperation unit relating to remote control on the application isprojected in a field of vision of a user wearing the glasses-typeinformation device by using the glasses-type information device, theremote control being executed by a linked device held by the user; andexecute function control of the application based on a command receivedfrom the linked device in accordance with a user operation performed bythe user on the linked device.
 2. The glasses-type information deviceaccording to claim 1, wherein the image corresponding to the operationunit is an image corresponding to a command received from the linkeddevice.
 3. The glasses-type information device according to claim 1,wherein the instructions further cause the glasses-type informationdevice to execute the display control of reflecting the user operationperformed on the linked device, in the projected image corresponding tothe operation unit, based on the command received from the linkeddevice.
 4. The glasses-type information device according to claim 1,wherein the instructions further cause the glasses-type informationdevice to provide the user with a projection setting for designatingwhether to project the image corresponding to the operation unit, in theremote control.
 5. The glasses-type information device according toclaim 4, wherein, in the projection setting, designating whether toproject the image corresponding to the operation unit is performable foreach application of the glasses-type information device.
 6. Theglasses-type information device according to claim 4, wherein, in theprojection setting, designating whether to project the imagecorresponding to the operation unit is performable in accordance with anenvironment detected with the glasses-type information device.
 7. Theglasses-type information device according to claim 1, wherein theglasses-type information device includes a binocular smart glass, amonocular smart glass, and a head-mounted display.
 8. The glasses-typeinformation device according to claim 1, wherein the glasses-typeinformation device includes therein a network module for directlyconnecting to the Internet.
 9. A non-transitory computer-readablestorage medium storing a computer program for causing a computer toexecute a method for a glasses-type information device configured toexecute an application, the method comprising: executing display controlin such a manner that an image corresponding to an operation unitrelating to remote control on the application is projected in a field ofvision of a user wearing the glasses-type information device by usingthe glasses-type information device, the remote control being executedby a linked device held by the user; and executing function control ofthe application based on a command received from the linked device inaccordance with a user operation performed by the user on the linkeddevice.
 10. The non-transitory computer-readable storage mediumaccording to claim 9, wherein the image corresponding to the operationunit is an image corresponding to a command received from the linkeddevice.
 11. The non-transitory computer-readable storage mediumaccording to claim 9, wherein the method further comprises executing thedisplay control of reflecting the user operation performed on the linkeddevice, in the projected image corresponding to the operation unit,based on the command received from the linked device.
 12. Thenon-transitory computer-readable storage medium according to claim 9,wherein the method further comprises providing the user with aprojection setting for designating whether to project the imagecorresponding to the operation unit, in the remote control.
 13. Thenon-transitory computer-readable storage medium according to claim 12,wherein, in the projection setting, designating whether to project theimage corresponding to the operation unit is performable for eachapplication of the glasses-type information device.
 14. Thenon-transitory computer-readable storage medium according to claim 12,wherein, in the projection setting, designating whether to project theimage corresponding to the operation unit is performable in accordancewith an environment detected with the glasses-type information device.15. The non-transitory computer-readable storage medium according toclaim 9, wherein the glasses-type information device includes abinocular smart glass, a monocular smart glass, or a head-mounteddisplay.
 16. The non-transitory computer-readable storage mediumaccording to claim 9, wherein the glasses-type information deviceincludes therein a network module for directly connecting to theInternet.
 17. A method for a glasses-type information device executingan application, the method comprising: executing display control in sucha manner that an image corresponding to an operation unit relating toremote control on the application is projected in a field of vision of auser wearing the glasses-type information device by using theglasses-type information device, the remote control being executed by alinked device held by the user; and executing function control of theapplication based on a command received from the linked device inaccordance with a user operation performed by the user on the linkeddevice.